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Abstract This article presents a content analysis of the 2013 School Leadership 
Program (SLP) grants. SLP projects provide a unique opportunity for participants in 
the field to explore innovative leadership preparation and development and their 
impact on program participants, schools, school districts, and students. The article 
begins with an overview of the SLP, the changing field of leadership preparation, and 
current research in the field. Findings then reveal a range of evaluation tools, meth¬ 
ods, and data, the presence of myriad evaluators participating in the projects, and 
little focus on external dissemination of program evaluation methods beyond the 
scope of the projects. Suggestions for research to extend the field are provided. 
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During the last decade, programs to prepare educational administrators have 
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undergone considerable change. Growing specialization in the field of 
educational administration resulting from new knowledge production 
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(for example, operations research) is one reason for the program change. 


Another is the continuing search for more effective patterns of field 


experience, instructional method, and content in preparatory programs. 


—Piele and Culbertson (Wynn, 1972, p. vii) 


In 1972, the Educational Resources Information Center (ERIC) and the University 
Council for Educational Administration (UCEA) published a monograph series on 
“administrator preparation.” Piele and Culbertson’s foreword (in Wynn, 1972) in each 
of the monographs in the series began with the quote above. This series focused on 
topics such as Unconventional Methods and Materials for Preparing Educational 
Administrators (Wynn, 1972), Training-in-Common for Educational, Public, and Business 
Administrators (Miklos, 1972), and Emergent Practices in the Continuing Education of 
School Administration (Lutz CrFerrante, 1972). Forty years later, the conversation con¬ 
tinues around how to best prepare school leaders and the “search for more effective 
patterns.” While field experiences, instructional methods, and content are still at the 
forefront of research about how best to prepare aspiring school leaders, the research 
field has expanded. Innovative programs, partnerships, inclusion of non-university- 
based leadership providers and delivery structures, as well as other contributing fac¬ 
tors that impact effective preparation, are being explored. 

Research on the effectiveness of school leadership preparation programs is sur¬ 
prisingly limited, in spite of the need to examine leadership preparation and its im¬ 
pact on the field. Recently, there have been numerous calls for more in-depth studies 
on the impact of those programs on school leaders and, ultimately, on student 
achievement (Browne-Ferrigno & Muth, 2012; Kottkamp & Rusch, 2009; Murphy 
& Vriesenga, 2006; Orr & Orphanos, 2011; Sanzo, 2012). The dearth of research 
on leadership preparation and development is even more shocking because of in¬ 
tense criticism in recent years about how school leaders have been prepared (Fless 
& Kelly, 2005; Levine, 2005), especially with the increased attention to and scrutiny 
of these programs in the wake of the scathing 2005 Levine report. Nevertheless, 
“there is a small and growing body of evaluation research on leadership preparation 
program models and features” (Orr & Barber, 2009, p. 458). 

Contributing to the accumulating evaluation research around leadership prepa¬ 
ration programs are evaluation findings from projects funded by the School 
Leadership Program (SLP) grant. Since 2002, the United States Department of 
Education (USDE) has funded 110 projects designed to prepare (and/or develop) 
aspiring and current assistant principals and principals. A component of the federal 
grants—to date totaling more than $170,000,000 in appropriations—is a require¬ 
ment to evaluate the intervention programs. There has been an increasing interest 
in the program evaluation components of these grants and the implications they 
may have on research into leadership preparation programs, including participant 
outcomes and impacts on student achievement. 


2 


IJEPL 11 ( 11 ) 2016 


This article extends an earlier analysis of select SLP projects, specifically around 
the program evaluation features in the funded proposals. In that paper, Sanzo (2012) 
examined fifteen funded proposals from the 2008, 2009, and 2010 grant award cy¬ 
cles. This study examines thirteen funded proposals from the 2013 USDE grant cycle. 
The decision to exclude the previous grants from this study (2008, 2009, 2010) was 
based on several factors. First, the grants had been previously reviewed in an earlier 
study. More significant, however, were the USDE’s revisions to the request for pro¬ 
posals for the 2013 grant cycle; these made it difficult to develop a longitudinal study 
using the same approach as was used for the first set of grants. Both the program 
features component and the evaluation components changed in significant ways. It 
was decided, therefore, to focus on this specific round of grants. This article begins 
with an overview of the SLP, discusses possible reasons behind the changing leader¬ 
ship preparation field, and shares research within the scope of leadership preparation. 
The process for analyzing the program evaluations in the 2013 funded proposals is 
explicated, followed by a presentation of the findings, implications, and suggestions 
for future research. 

Significance 

A review and analysis of these program evaluation proposals can help to extend the 
field’s knowledge of the types of measures, tools, processes, and techniques recog¬ 
nized by national experts in the field and being used by disparate organizations. It 
is important to know what methods of evaluation are being used and how these 
methods impact our ability to know (or not know—depending on their limitations) 
what effect the programs actually had. This analysis provides insight into which crit¬ 
ical program evaluation approaches the USDE recognizes as worth funding. 
Knowledge of this can then translate to a streamlined integration of evaluation efforts 
not only across SLP projects, but also within the broader school leadership prepara¬ 
tion community. The SLP grant program represents the unique and concerted efforts 
of leadership preparation and development programs across the United States to 
shape and advance the state of leadership preparation. Not understanding what is 
actually occurring in these SLP projects would mean missing prime opportunities 
to truly understand the real impact of leadership preparation on the field. This is 
not a new argument. Kottkamp and Rusch (2009) wrote that what “we as a field of 
research produce is a lot of islands sprinkled across a vast sea” (p. 80). It is incumbent 
upon us now, as researchers and scholars, to bring these islands together to create a 
large enough mass to demonstrate effectiveness and impact in a concerted manner. 

Setting the context: The SLP 

At the federal level, the United States Department of Education (USDE) has spon¬ 
sored the School Leadership Program (SLP) grant since 2002 to fund innovative lead¬ 
ership preparation and development programs. According to the USDE: 

The School Leadership Program provides competitive grants to as¬ 
sist high-need local educational agencies (LEAs) with recruiting, 
training, and retaining principals and assistant principals. A high- 
need LEA is defined as one that: (1) either serves at least 10,000 
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children from low-income families or serves a community in which 
at least 20 percent of children are from low-income families; and 
(2) has a high percentage of teachers teaching either outside of their 
certification or with emergency, provisional, or temporary certifica¬ 
tion. (USDE, n.d.) 

The grant policy changed in 2008 to fund five-year programs, in comparison with 
the previously three-year program cycles, in direct response to the need to increase 
the time frame in which to measure grant effectiveness. A greater awareness of impact 
and outcome measures has guided conversations around these grants in recent years. 
The 2011, 2012, and 2013 national conferences sponsored by the School Leadership 
Program Communication Hub (now School Leadership Preparation and 
Development Network [SLPDN]) dedicated a significant amount of time to program 
evaluation and the impacts of the program features. The SLPDN is an organization 
that connects all current and past SLP grantees in a variety of ways, including 
through national conferences, webinars, and collaborative research and publications. 

Characteristics of these grants are innovative approaches to leadership prepara¬ 
tion and development that go beyond university-based classroom instruction that is 
typically separate from district leadership practices. Each grant involves a variety of 
partnerships to design and implement activities to prepare aspiring leaders and/or 
provide professional development to current assistant principals and principals. 
Examples of innovative activities described by the USDE (USDE, n.d.) include fiscal 
incentives to retain new leaders and to encourage others to enter school leaders, pro¬ 
viding stipends for mentors, and professional development that targets instructional 
leadership. 

Leadership preparation: Criticism and change 

Criticism of leadership preparation programs is not new. The 1972 series on admin¬ 
istrator preparation referenced at the beginning of this manuscript cited concerns 
about preparation programs that, in some cases, parallel contemporary critiques. 
Structural components were found to lack “a Gestalt conception” (Farquhar & Piele, 
1972, p. 16) programs suffered from “limitations and obstacles that hinder recruit¬ 
ment” (p. 21), and some participants expressed “little satisfaction with the internship” 
(p. 31). Similar concerns associated with leadership preparation can be seen through¬ 
out subsequent decades. In 1987, UCEA sponsored a blue-ribbon panel focused on 
developing administration skills. This panel’s report identified problems with lead¬ 
ership preparation which included “the poor quality of candidates for preparation 
programs” and “the irrelevance of preparation programs; programs devoid of se¬ 
quence, modern content and clinical experiences” (Hale & Moorman, 2003, p. 6). 

“Traditional” notions of leadership preparation remain and continue to be fodder 
for unfavorable critiques with researchers, policymakers, and practitioners. In 2004, 
the U.S. Department of Education’s Office of Innovation and Improvement (Oil) 
blamed “traditional education administration programs and certification procedures” 
for “producing insufficient numbers of (educational) leaders” (U.S. Department of 
Education, Office of Innovations and Improvement, 2004, p. 3). Further, the Oil 
faulted these programs for not having a recruitment process and instead allowing 
students to self-enroll, for having insufficient screening processes for applying stu- 
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dents, for failing to have an interconnected program curriculum, for having poor 
linkages between theory and practical application, and for not connecting the pro¬ 
gram to the individual’s projected career path and administrative placement (i.e., 
rural or urban setting) (U.S. Department of Education, Office of Innovation and 
Improvement, 2004). 

Notably, Levine’s 2005 report, Educating School Leaders , condemned leadership 
programs for being outdated and for participating in a “race to the bottom.” This re¬ 
port served as a polarizing call within the field of educational leadership preparation. 
His report perhaps best crystallizes the “traditional” notion of leadership preparation. 
Levine (2005) found rampant “curricular disarray,” “low admissions and graduation 
standards,” and “inadequate clinical instruction,” among other cutting assessments. 
Other critics have expressed alarm about the state of leadership preparation, includ¬ 
ing the state of research in the field: 

Given the applied nature of the profession and the centrality of 
preparatory activities to departments of educational leadership, the 
fact that serious academic work on pre-service training remains a 
minor element in the school administration scholarship mosaic is as 
surprising as it is disappointing. (Murphy & Vriesenga, 2006, p. 187) 

More recently, a focus on such elements of leadership preparation as school 
leader evaluation and the relationship between leaders and student achievement 
have gained momentum (Sanzo, 2014; Sanzo, 2016). The increased interest in the 
role of the school principal in relation to student achievement has accelerated the 
proliferation of non-university-based preparation providers in the field and increased 
the influence of foundations on training practices. 

State and national policies around school leadership and leadership preparation 
programs have changed in turn. Examples of these changes include the 2015 col¬ 
laboration of a consortium of national organizations and policymaking bodies to re¬ 
vise the national leadership standards to create the Professional Standards for 
Educational Leaders (National Policy Board for Educational Administration, 2015); 
state mandates for massive overhauls of university-based leadership programs in 
states such as in Kentucky and Alabama; national attention to the evaluation of 
school principals and the connection between leadership effectiveness and student 
achievement; a proliferation of value-added evaluation models for principal evalua¬ 
tion at the state level; and the requirement, from the Council for Accreditation of 
Educator Preparation (CAEP), that programs demonstrate the impact of their candi¬ 
dates on student learning (National Policy Board for Educational Administration, 
2015). Additionally, UCEA has developed research briefs, conceptual guides, and 
other materials to aid in leadership preparation and development programs, such as 
A Policymakers Guide: Research-Based Policy for Principal Preparation Program Approval 
and Licensure (Anderson & Reynolds, 2015) and The State Evaluation of Principal 
Preparation Programs Toolkit in collaboration with New Leaders (UCEA & New 
Leaders, 2016). 
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the critics (Sanzo, 2014). This is an era of greater accountability and transparency, 
with calls for linkages between programs and the impact of their graduates in the 
schools they serve (Kowalski, 2009). Additionally, the role of school leaders has 
changed dramatically. University faculty, not-for-profit foundations, and other stake¬ 
holders interested in preparing school leaders have contributed to the increasing 
body of work in the field. The Danforth Foundation, for example, responded in the 
1980s and 1990s with the funding of the Danforth Programs for the Preparation of 
School Principals (Milstein, 1993) and conducted case studies on their program sites. 
More recently, the Wallace Foundation began funding projects and research on 
school leadership, totaling about $286 million since 2000. And the recent call from 
the United States Department of Education for school leadership program proposals 
has highlighted the need for evidence-based research to support the programs. 

More research on leadership preparation programs is emerging from program eval¬ 
uations (see Aitken & Bedard, 2007; Buskey & Karvonen, 2012; Davis & Darling- 
Hammond, 2012; Eddins, Kirk, Hooten, & Russell, 2013; Haeffele, Hood, CrFeldmann, 
2011; Sanzo, 2012). Orr and Barber (2009) deftly synthesized the prevailing thoughts 
of and trends for program evaluation in leadership preparation and relied on Rossi, 
Freeman, and Lipsey’s (1999) definition of program evaluation as being 

the use of social research methods to systematically investigate the 
effectiveness of social intervention programs. It draws on the tech¬ 
niques and concepts of social science disciplines and is intended to 
be useful for improving programs and information social action. 

(cited in Orr & Barber, 2009, p. 457) 

Orr and Barber (2009) highlight the political tensions that can be involved in eval¬ 
uating programs, especially in light of the evaluation’s ability to “influence resource 
allocations, institutional decisions, and policy and program reform” (p. 458). 

Other examples of increased research in the field include the development of 
the Journal for Research of Leadership Education by UCEA. This publication was de¬ 
veloped in 2006 and provides cutting edge research on the preparation of aspiring 
and current leaders. The Taskforce on Evaluating Leadership Preparation Programs 
also contributed to the increased knowledge base on school leadership preparation 
programs as well. The Taskforce began as a pre-session conversation at the 2001 
UCEA conference (Kottkamp & Rusch, 2009). Since that meeting, the Taskforce ex¬ 
panded its role and provided “an example of different use of time and of the potential 
to grow cross-institution research communities” (Kottkamp & Rusch, 2009, p. 81). 
In collaboration with the Utah Educational Policy Center, UCEA also supports the 
Center for the Evaluation of Educational Leadership Preparation and Practice. In 
2013, the center presented an updated version of the 2010 Developing Evaluation 
Evidence: A Formative and Summative Evaluation Planner for Educational Leadership 
Preparation Programs (Orr, Young, & Rorrer, 2010). 

Program characteristics, or “features,” have garnered a large amount of interest 
and research focus in an effort to identify how to effectively prepare aspiring school 
leaders. Most research has involved “case studies of innovative preparation programs 
and survey-based investigations of the efficacy of specific program features” (Orr & 
Orphanos, 2011, p. 21). “Innovative” program features have been recognized as 
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being more effective. Such features include a well-defined theory of action, a coherent 
curriculum in alignment with national and state standards, integration of theory and 
practice, and socialization and support mechanisms such as cohorts and mentoring 
(Orr & Orphanos, 2011). 

Recent attention has been given to the effectiveness of programs for leadership 
preparation (Belle & Sanzo, 2014; Burt, Shen, Leneway, & Rainey, 2014; Salazar, 
Pazey, & Zembik, 2013; Sanzo, 2016). For example, Fuller, Young, and Baker (2011) 
and Orr and Orphanos (2011) examined the influence of principal preparation pro¬ 
grams on student achievement. Orr (2011) studied the program experiences and ca¬ 
reer outcomes of graduates of leadership preparation programs. It appears the trend 
of exploring program features may be giving way to more focus on the impacts of 
those features on program graduates, the schools they lead, and the achievement of 
their students. This makes the SLP program especially important to study because 
of the programs attention to the short-, mid-, and long-term impact of innovative 
programs on student achievement, graduates, and the schools and districts the grad¬ 
uates serve. 

Methods 

A content analysis was conducted on selected funded 2013 SLP proposals. The re¬ 
search questions draw from the Sanzo (2012) study and were modified based upon 
the results of that analysis, as well as on a review of the current literature: 

1. What evaluation methods are proposed in the SLP grant applications? 

2. What evaluation tools are proposed in the SLP grant applications? 

3. What are the data being collected and how are they analyzed? 

4. Who are the evaluators? 

5. What is the purpose of the program evaluation? 

Twenty grants were funded in the 2013 cycle. The focus of this article is on lead¬ 
ership preparation, rather than on the professional development of current school lead¬ 
ers. Therefore, six of the proposals were excluded from the review because of their 
sole focus on the development of already licensed school leaders. A seventh proposal 
seemed to focus, in part, on preparing aspiring leaders, but it was difficult to under¬ 
stand whether the grant activities actually focused on the preparation component, and 
therefore it was excluded. The remaining thirteen focused either solely on preparing 
aspiring school leaders or on both preparation of aspiring school leaders and providing 
professional development to current assistant principals and principals. 

A content analysis was employed on the USDE SLP grants (Patton, 2002). A re¬ 
search team familiarized themselves with the coding schema used in the 2012 Sanzo 
study (Appendix A) and read through each grant proposal first, then through the 
entire funded grant proposal. The process allowed the researcher and the team mem¬ 
bers to become familiar with the program evaluations throughout all proposals, and 
then understand the broader context within which the proposals were situated. 
Because of the number of proposals reviewed, the research team employed a manual 
coding process, rather than an electronic program. This allowed the research team 
to develop more familiarity with the program evaluation proposals. During this 
process, notations were made on the original analysis document (Sanzo, 2012) wher- 
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ever revisions, additions, and deletions were necessary because of the new round of 
funding. As in the original study, the research team used the following process: “Each 
program evaluation section was initially reviewed in the selected grant proposals, 
followed by an overall review of each complete grant application. This process al¬ 
lowed the researcher to identify the specific information required from each grant 
application that would be needed for a more detailed analysis to address the research 
questions” (Sanzo, 2012, p. 249). The analysis process included the grant abstract, 
the grant proposal in its entirety, and specifically the program evaluation and man¬ 
agement plan. Although the appendices are not part of the consideration process for 
funding by the USDE, the team felt this was an important component to review, as 
a further window into the program evaluations. Throughout the process, the research 
team took notes to create a preliminary coding schema, through which a coding 
guide was developed (Miles & Huberman, 1994). 

After discussion among the research team, a decision was made to collapse the 
themes relating to methods, data, and tools into one category, primarily because of 
the relational ambiguity of each as well as the overlapping references in the grants. 
Because of the involvement of the author in a previous study on the same topic, the 
author sought to maintain transparency and ethicality by implementing triangulation, 
peer debriefing, reflective commentary, and member checks (Harrison, MacGibbon, 
& Morton, 2001; Schwandt, Lincoln, & Cuba, 2007): 

Throughout this process, the researchers ask participants if the 
themes or categories make sense, whether they are developed with 
sufficient evidence, and whether the overall account is realistic and 
accurate. (Creswell & Miller, 2000, p. 127) 

Sanzo, in the 2012 content analysis of program evaluations, argued, “It is neither 
the purpose nor intent of this manuscript to evaluate the quality of the leadership 
preparation program proposal, nor to make a judgment on the quality of the evalu¬ 
ation” (p. 247). This study adds to the baseline understanding first begun by Sanzo 
(2012) about the various facets of the funded program evaluation proposals, includ¬ 
ing designs, methods, personnel, and tools: “With better understanding about the 
nature, processes, and content of the program evaluations, further research on lead¬ 
ership preparation programs can build on this foundation” (Sanzo, 2012, p. 248). 

Limitations 

This study is an analysis of thirteen funded grant proposals that are a part of the 
twenty funded proposals for the 2013 USDE SLP cycle, and the findings are limited 
to those grants. Most of the proposals that focus on aspiring leadership preparation 
also include program features that focus on providing professional development to 
current school leaders. Wherever obvious in the proposal, the evaluation measures 
related to current school leaders were excluded from the analysis. However, because 
of the design of the proposals, it was difficult to bifurcate all their components, so 
some conflation of aspiring leadership program design and current leadership de¬ 
velopment program design may have occurred. 

These proposals also are just that—proposals. The funded projects have the lee¬ 
way to make revisions and are encouraged to do so as the project is implemented, 
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with USDE approval. It may be the case that certain components were revised, elim¬ 
inated, or added in subsequent follow-ups about implementation of the program 
evaluation. However, the intent of this study is to understand the components of 
the funded program evaluations. 

Finally, the review only included what was publicly available through the 
Freedom of Information Act (FOIA). The materials provided through the request 
did not include any appendices and only comprised the fifty or fewer pages in the 
formal application. While the funders’ explicit directions were that any items in¬ 
cluded in the appendices would not be a part of the reviewed proposal, it appears 
there were appendices in some proposals that further explicated the program evalu¬ 
ation (i.e., timelines and personnel). Therefore, the research team may not have had 
access to all of the content of the proposed program evaluations. 

Findings 

The results of the content analysis are presented below. As noted above, the analysis 
was conducted on the funded proposals made available through the FOIA request. 
As evidenced in the presentation of the findings, and as referenced earlier, it was 
clear that additional information related to the program evaluations had been pro¬ 
vided in the grant proposal packet provided to the funding agency. Additionally, the 
program evaluations only reflected what was developed for the purpose of the pro¬ 
posal and may have been modified upon funder approval to meet the evolving needs 
of the grant. There was a range of methods, tools, data and overall ambiguity in 
many of the program evaluations; there were myriad evaluators anticipated to be 
utilized in the program evaluations; and program evaluations were geared primarily 
toward the grant itself. 

Ambiguity: Methods, tools, and data 

Included in nine of the proposals was some type of logic model used to inform both 
the program implementation and the program evaluation. It was apparent that grants 
included additional information such as in-depth timelines, but these were not made 
available by the FOIA request, nor were they a part of the official review by the USDE 
as outlined in the request for proposals. Therefore, it was not possible to review 
these specific tools. 

Each proposal referenced the use of quantitative and qualitative data collection 
procedures. There was very little description of the types of methods and tools that 
would be used for the collection of qualitative data in the program evaluations. 
Examples of the vagueness of approaches and data types include 

• observation of key leadership activities and venues 

• assessment of quality of co-operation of school districts and 
quality of inter-institutional collaboration related to delivery of 
common core instruction 

• participants’ self-assessment of strengths and weaknesses 

• site visits 

• ongoing benchmarks for culture improvement (specific to each 
school) 
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• quality of peer and principal teacher observations 

• ongoing leadership interviews by project staff 

• coaching logs 

• qualitative notes from school leadership interviews 

• focus groups 

• surveys (perceptual and attitudinal) 

Only limited qualitative tools were specifically identified in the proposals. One 
project cited the use of the “instructional rounds” process developed by Richard 
Elmore and his colleagues and another identified a specific tool called the 
“Leadership Cohort Exit Survey” which provided qualitative data. One grant also 
referenced the use of both the EDM Quality Measures tool and the UCEA Program 
Quality Assessment. These two tools use a rubric system to evaluate the effectiveness 
of leadership preparation programs. 

In terms of quantitative data, most often the long-term effectiveness of the proj¬ 
ects was measured through student achievement data. Primarily student achievement 
data was being measured through state assessments or general student achievement 
measures. Evaluation of the data ranged from simple comparisons of the data to mul¬ 
tilevel modeling and propensity score matching. Additionally proposals referred to 
the analysis of data through the use of descriptive processes, comparisons, and score 
trend analysis without any further descriptions. 

Few grant proposals proffered specific quantitative tools that would be used in 
the grant. References were made to developing reliable and valid tools to use in the 
grant. The VAL-ED was referenced twice, the Chicago 5 Essentials support once, with 
the use of the SLLA exam as measures of success in other program evaluations. While 
grant proposals indicated they would use the following data in their evaluations, they 
provided no specifics: school level aggregate data, attendance, tardiness, discipline, 
school climate, academic rigor, graduation rates, pre-post scores (often with no addi¬ 
tional description), 360 resident evaluation, program evaluator data sources, and GREs. 

In sum, the methods, data, and tools were generally vague, with little description 
of what types of data would be sought, how they would be collected, or how they 
would be analyzed. 

Myriad evaluators 

There was a mix of evaluators who were identified as participating in the grant. 
Eleven of the thirteen proposals identified the program evaluators for the projects. 
Seven projects employed external evaluation companies and one of the projects used 
an external consultant (single person). Three grant proposals identified evaluators 
internal to the project, with one grant using one person to evaluate the grant and 
two projects using a lead evaluator and a team to assist. Finally, two of the grant pro¬ 
posals indicated they would contract with an external evaluator after the awarding 
of the grant from the USDE, meaning ultimately nine of the grants would have eval¬ 
uators who were not one of the project partners providing preparation and/or devel¬ 
opment services. 

Three of the external evaluator companies had also served as external evaluators 
on previously awarded USDE SLP projects. These external evaluators are large, na- 
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tional evaluation companies that are prominent in evaluation projects related to ed¬ 
ucation. One of these external companies is listed as the program evaluator for two 
of the 2013 funded projects reviewed for this study, meaning the three external eval¬ 
uators served four funded projects in total. 

Interestingly, those same projects made up four of the five projects using specific 
research questions (in addition to the project goals and objectives) to guide their 
evaluations. The fifth project with research questions employed an internal evaluator 
with significant experience in program evaluations, having previous evaluation ex¬ 
perience with large-scale grants. 

Program evaluation intent: Primarily grant focused 

There was one grant that discussed a specific approach, utilization-focused evalua¬ 
tion (Patton, 2002), from the field of program evaluation. There was no reference to 
the use of evaluation tools or processes from any of the UCEA centers, outside of 
the reference to the “program quality” rubric. Additionally, no grants referenced spe¬ 
cific program theories to inform the evaluation model. 

Twelve of the thirteen grants specifically talked about the dissemination of proj¬ 
ect findings. Methods of dissemination were vague—generally publications and con¬ 
ference presentations. None of the dissemination efforts focused directly on the 
utilization of evaluation processes and lessons learned from an evaluative standpoint. 
All dissemination efforts were directed toward leadership development. A compo¬ 
nent of the grant proposal was to include the potential contributions to the field. 
This was reviewed in each of the funded proposals to see if information regarding 
evaluation methods was considered to merit as a contribution to the field. In none 
of the proposals reviewed was this seen as a potential contribution to the field. While 
most of the projects sought to contribute to the broader field of leadership prepara¬ 
tion and development, this was specifically for program features and not related to 
the program evaluation methods. 

Discussion, implications, and future research 

This study extends the conversation about program evaluations developed for use in 
the USDE School Leadership Program grant (Sanzo, 2012). In this current study, the 
2013 analysis revealed similar findings as in Sanzo (2012). The discussion of evalua¬ 
tion methods and tools was limited, and the majority of the evaluators identified in 
the proposals ranged from externally contracted personnel and companies to internal 
grant personnel. This study also reveals that while there was much interest on the 
part of the grant writers/team in disseminating lessons learned about the projects, 
there was no articulated interest in specifically discussing the program evaluation 
methods separately from the findings. This section will provide a discussion on the 
above findings and their implications and will make suggestions for future research. 

The findings may, on one hand, suggest a cause for concern because of the lack 
of detail about the overall measures, tools, and general approach to the program 
evaluations; on the other hand, the findings may be encouraging. It is evident from 
the review of the proposals that the grant writing teams worked together to develop 
a comprehensive and integrated program that deeply connected the various program 
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features and the project measures for the program evaluation. The program evalua¬ 
tions did not present themselves as generic evaluations; rather, it appeared that feed¬ 
back from the program evaluations was intended to provide guidance to the grant 
project personnel to make ongoing changes throughout the course of the project. 
Fifty pages is not a lot of space to detail a very large five-year project. Critics may 
argue that the lack of specificity is a detriment to the project, that it can potentially 
impede the successful evaluation of the overall program, and that it might adversely 
impact participants, districts, and PK-12 students. However, at the same time, the 
ambiguity provides a measure of flexibility—to be more “design-based” in thinking 
about program evaluation, to potentially develop real-time measures and tools with 
direct application to the project, and to implement the lessons learned from those 
evaluative tools and measures. The USDE has expressed an understanding that pro¬ 
gram designers may need to make ongoing revisions both to the program features 
and the program evaluation components; this ambiguity at the proposal phase pro¬ 
vides that opportunity, rather than locking a program into a five-year, unchangeable 
initiative. It is difficult to predict how a leadership development program might 
change over the course of five years. Each program evaluation’s approach appears to 
provide the flexibility to adapt to evolving programs. 

While it is encouraging that there is more potential research emerging from pro¬ 
gram evaluations, as Orr and Barber (2009) point out, more effort is required to 
bring lessons learned from program evaluations into the field of leadership prepara¬ 
tion. School leadership preparation research and the evaluative methods used by in¬ 
dividual leadership development programs can be better aligned. Bridging the gap 
between program evaluation and research is critical. It appears from the content 
analysis that SLP projects want to share the lessons learned through their program 
evaluations with the broader leadership preparation and development community. 
However, what is less clear is the intent to share the methodological approaches of 
their evaluations with the field. This is a significant limitation and harms the field if 
there is not a specific discussion around program evaluation methods and around 
how evaluations for programs are revised across the duration of the projects. 
Emerging studies from the SLP grants provide a venue for a focus on evaluation and 
impact, as well as specific program features. 

An argument can be made that, with today’s push for rigorous research, there is 
a concern that the field of educational preparation is not looking more deeply into 
disseminating their program evaluation methods. Yet it is probable that this is an in¬ 
dication that grant members involved in the work are incredibly busy and not able 
to focus their attention on multiple dissemination efforts. It is incumbent on the 
broader field of educational leadership scholars to contribute to the work and to par¬ 
ticipate in collaborative research that capitalizes on program evaluations’ findings to 
improve projects and to disseminate these findings on leadership preparation and 
development programs. The funded proposals do highlight a desire to publish les¬ 
sons learned and effective program features, and this will help share the types of 
methods employed in the evaluations. 

The methods and tools provided in the grant proposals were limited. It may be 
the case, as mentioned earlier, that this was due to the page limits imposed on the 


Sanzo 

Content Analysis 
of 2013 
SPL Evaluation 
Proposals 


12 


IJEPL 11 ( 11 ) 2016 


applicants. However, if there is an interest in methods, as articulated by the field’s 
call for improvement in preparation and development practices, then programs must 
better understand evaluation techniques. Most of the grants made allusions to survey 
tools and other instruments without specifying the types of tools they would choose 
to employ. The SLP grant has been in place for over a decade and there is enough 
historical perspective to understand the types of tools and methods that have been 
used. While Sanzo’s (2012) study is the first to explore the SLP grant proposals, other 
studies have been published featuring specific SLP grant findings, including the types 
of program evaluation methods that were used (Sanzo, 2014; Sanzo, Myran, & 
Normore, 2012). If it is the case that there is no broad understanding of these tools, 
then it is imperative to catalogue these, to identify their best uses and possible con¬ 
straints, and to share these with the field. Additionally, future research is recom¬ 
mended around the fidelity of program evaluation implementation by different 
configurations of evaluation teams, including the number of team members, the 
background and experience of the team members, and whether the evaluators are 
internal or external to the grant. 

The SLP grant provides a unique opportunity for the field to explore innovative 
leadership preparation and development and their impact on program participants, 
schools and school districts, and students. This article provides only an initial un¬ 
derstanding of one small facet of the overall program. It is incumbent on us as a field 
to coalesce the “islands sprinkled across a vast sea” (Kottkamp & Rusch, 2009) in 
order to create a large enough mass for research purposes and in turn to impact the 
field of leadership preparation in a large-scale and focused manner. 
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Partners 


Evaluation Approach 


• Lead Partner 

• Other 
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• Articulated evaluation theory 

• Articulated methodological approach 

• Articulated purpose 
Evaluation Tools 

• Empirically validated and tested 

• Developed during the grant by the evaluators for the grant 

• Developed by a third party vendor 

• Formative use 

• Summative use 

• Local consumption use 

• providing data intentionally transferable to other leadership 
prep programs 


Data 


• Quantitative 

• Qualitative 

• Project participants (aspiring leaders, current leaders, coaches, 
mentors, other) 

• Timeline for data collection 

• Articulated means of analysis 


Evaluator 


• Internal to lead grantee 

o private company 
o university-based group 
o university-based person 
o district-based 
o private person 

• Internal to one of the project partners 

o private company 
o university-based group 
o university-based person 
o district-based 
o private person 

• Third party vendor/External 

o private company 
o university-based group 
o university-based person 
o district-based 
o private person 


16 


